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CLAIMS 

1 . A system that enables a user to interact with a virtual con- 
trol panel using a user controlled pointing object, wherein the 
5 system comprises a first tracl<ing unit adapted to capture data 
representing the position of the pointing object, characterized in 
that the system further comprises: 

- a portable identification element, 

- a second tracking unit adapted to capture data representing 
10 the position of the identification element, 

- a storage unit, storing at least one pre-defined graphical in- 

terface representing a control panel of a device, said 
graphical Interface comprising an interface for user Inter- 
actions with the device, 
15 - a graphics unit, generating a graphical representation of the 

control panel based on said stored graphical interface, 

- a registering unit, registering said graphical representation 

pf the control panel in a fixed relation to said portable 
identification element, based on said data representing 
20 the position of the identification element, to produce a 

virtual control panel, 

- a display unit, showing the user a view comprising the real 

world and the virtual control panel projected In a fixed re- 
lation to said portable identification element, and 
25 - an application unit, performing actions in response to the 

users interactions with the virtual control panel, and de- 
termining which actions to be performed based on the 
position of said user controlled pointing object In relation 
to the identification element. 

30 

2. A system according to claim 1, character ized In that the 
system is adapted to modify said graphical representation of the 
control panel in response to interactions between the user con- 
trolled pointing object and the virtual control panel. 

35 
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3. A system according to claim 1 or 2. characterized in that 
said graphical Interface is adapted for displaying data from the 
device and that the system is adapted to generate a graphical 
representation of the data and to display the data on the virtual 

5 control panel, In response to interactions between the user con- 
trolled pointing object and the virtual control panel. 

4. A system according to any of the previous claims, charac- 
terized in that said user controlled pointing object Is a handheld 

10 pointing device or a part of the user's body. 

5. A system according to any of the previous claims, charac- 
terized in that the storage unit is adapted for storing a plurality 
of graphical Interfaces, each representing a control panel of a 

15 particular device, that the system is adapted to generate and 
display a plurality of graphical representations of control panels 
for different devices based on said stored graphical interfaces of 
the devices and that the system comprises means for Identifying 
which of the stored control panels to be displayed. 

20 

6. A system according to claim 5, characterized in that said 
means for identifying which of the stored control panels to be 
displayed comprises a recognition unit for recognizing and iden- 
tifying devices in the environment of the user and that the 

25 graphics unit is adapted to generate a graphical representation 
of the control panel of an identified device based on the stored 
graphical interface of the identified device. 

■ 

7. A system according to claim 6, r.haracterized In that said 
30 recognition unit is adapted for recognizing and Identifying 

unique identification markings on the devices. 

8. A system according to any of the claims 5-7, characterized 
in that the system is arranged so that it automatically changes 

35 the virtual control panel displayed when another device is rec- 
ognized and identified. 
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9. A system according to any of the previous claims, charac- 
terized in that said portable identification element is adapted to 
be carried by the user during interaction with the virtual control 

5 panel. 

10. A system according to any of the previous claims, phgrac- 
terized in that said portable identification element is attachable 
to the body of the user. 

10 

11. A system according to any of the previous claims, charac- 
terized in that said display unit comprises a head mounted dis- 
play showing the user said view. 

15 12. A method that enables a user to interact with a virtual con- 
trol panel using a user controlled pointing object, wherein the 

method comprises: 

- receiving data representing the position of the user con- 

trolled pointing object, 
20 - receiving data representing the position of a portable identi- 
fication element, 

- storing at least one pre-defined graphical interface repre- 

senting- a control panel of a device, 

- generating a graphical representation of the control panel of 
25 the device based on said pre-defined graphical Interface, 

- registering said graphical representation of the control 

panel in a fixed relation to said portable identification 
element, based on said data representing the position of 
the identification element, to produce a virtual control 

30 panel, 

- displaying a view comprising the real world and the virtual 

control panel projected in a fixed relation to said portable 
identification element, and 

- performing actions in response to the users Interactions 
35 with the virtual control panel, wherein the actions to be 

performed is determined based on the position of said 
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user controlled pointing object in relation to the position 
of the virtual control panel. 

13. A method according to claim 12, comprising defining a two- 
5 way communication between the virtual control panel and the 
device, and generating said graphical representation of the con- 
trol panel based on interactions between the user-controlled 
pointing object and the virtual control panel. 

10 14. A method according to claim 12 or 13, wherein it comprises, 
receiving data from the device, generating a graphical represen- 
tation of the received data and displaying the data on the virtual 
control panel. 

15 15. A method according to claim 14, wherein said data is dis- 
played on the virtual control panel in response to interactions 
between the user controlled pointing object and the virtual con- 
trol panel. 

20 16. A method according to any of the claims 12-15, wherein 
the method comprises: 

- storing a plurality of pre-defined graphical interfaces, each 

representing a control panel of a particular device, 

- determining which of the stored control panels to be dis- 
25 played, and 

- generating a graphical representation of the control panel to 

be displayed based on the pre-defined graphical inter- 
face of the control panel. 

30 17. A method according to claim 16, wherein it comprises rec- 
ognizing a device, determining which of the stored control pan- 
els to be displayed based on which device is recognized and 
generating a graphical representation of the control panel of the 
recognized device based on the stored graphical interface of the 

35 recognized device. 
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18. A method according to claim 17 characterized in that the 
devices are provided with unique identification markings and a 
device is recognized by identifying its unique identification merit- 
ing. 

5 

19. A method according to any of the claims 17-18, wherein 
the virtual control panel displayed is automatically changed 
when another device is recognized. 

10 20. A method according to any of the claims 12-19, wherein 
said portable identification element is carried by the user during 
interaction with the virtual control panel. 

21. A method according to any of the claims 12 - 20, wherein 
15 the virtual control panel comprises virtual interaction members 

and an audio feedback is generated when the user activates any 
of the virtual interaction members. 

22. A computer program comprising program instructions which, 
20 when loaded into a computer, causes the computer to perform 

the process of any of the claims 12-21. 

23. A computer readable medium having a program recorded 
thereon, where the program is to make a computer perform the 

25 steps of any of the claims 12 -21, when said program is run on 
the computer. 
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